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REMARKS 

In response to the objection to the drawings, Applicants have submitted 
herewith a replacement sheet containing the single figure, which has been 
enlarged, and which has been amended by the insertion of reference numerals, 
as required. In addition, the Abstract of the Disclosure has been amended to 
delete the reference to "[Fig. 1]" in line 7, and the revised Abstract is attached 
hereto on a separate sheet, as required. Finally, the specification has been 
amended to insert a reference to the priority document, as suggested in item 6 on 
page 3 of the Office Action, and appropriate headings have been inserted as well. 
Accordingly, reconsideration and withdrawal of the grounds of objection set forth 
in items 3 through 6 are respectfully requested. 

In response to the Examiner's suggestion in item 7 on page 4 of the Office 
Action, as well as the objections to Claims 7 and 12 as set forth in items 8 and 9 
of the Office Action, Applicants have amended the claims in the manner 
suggested by the Examiner. Accordingly, reconsideration and withdrawal of 
these grounds of objection are also requested. 

Claims 1, 2, 8-12 and 15 have been rejected under 35 U.S.C. §102(e) as 
anticipated by Watanabe (Published U.S. Patent Application No. 2003/0048926 
Al); while Claim 3, 4, and 7 have been rejected under 35 U.S.C. §103(a) as 
unpatentable over Watanabe in view of Schwerdt et al ("Visual Recognition of 
Emotional States"); Claim 13 has been rejected as unpatentable over Watanabe 
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in view of Kamin (U.S. Patent No. 4,198,653); Claim 14 has been rejected as 

unpatentable over Watanabe in view of Wyschogrod et al (U.S. Patent No. 

5,374,932); Claim 5 has been rejected as unpatentable over Watanabe and 

Schwerdt et al as applied to Claim 3, and further in view of Zuniga (U.S. Patent 

No. 5,546,474); and Claim 6 has been rejected as unpatentable over Watanabe 

and Schwerdt et al as applied to Claim 3, and further in view of Jepson et al 

(U.S. Patent No. 7,058,205). However, for the reasons set forth hereinafter, 

Applicants respectfully submit that all claims of record in this application 

distinguish over the cited references, whether considered separately or in 

combination with other references. 

The present invention is directed to a method and apparatus for 
processing video images for the purpose of generating an alarm signal when an 
"event of interest" is detected. More particularly, the invention provides a 
technique for detecting anomalous or "abnormal" motion in video images by 
identifying point objects therein, learning normal behavior patterns for the 
identified point objects, and then detecting abnormal behavior of the point 
objects by comparing their current behavior with the learned normal behavior 
patterns. For the latter purpose, according to the present invention, the position 
and movement of each extracted point feature within the video image is tracked, 
and the result of the tracking over time (that is, in successive images) is used in 
an iterative learning process to derive a normal pattern of behavior for the track 
in question. Subsequently, the behavior of the track in question is compared to 
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the normal pattern of behavior learned for that particular track, and an alarm is 

generated if the present behavior falls outside the normal pattern. 

The latter features of the invention are recited in Claim 1, which defines 
"a method for processing video images to detect an event of interest" that 
includes the steps of 

"using an iterative learning process to derive a normal pattern 
of behavior for each track; 

comparing present behavior of the at least one track to the 
respective normal pattern of behavior; and 

in response to the present behavior falling outside the normal 
pattern of behavior, generating an alarm signal". 

The present invention (as defined in Claim 1, for example) has a number of 
advantages relative to prior art video monitoring systems, such as video motion 
detectors and the like. In particular, the system according to the invention is 
able to detect small objects more effectively. In addition, it is also able to 
distinguish between objects that are moving in a learned "normal" fashion, and 
objects that are moving anomalously. Furthermore, unlike previous systems, it 
is relative insensitive to changes in seen illumination levels. Finally, it is also 
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simple and inexpensive to implement. (See page 2, lines 1-4; page 5, lines 22-29; 

page 7, line 18 — page 8, line 11; and page 16, lines 16-24.) 

The primary Watanabe reference, on the other hand, discloses a 
surveillance method and apparatus for detecting the presence of particular 
persons and identifying them, based on stored information, referred to as a 
"personal behavior table 15". For this purpose, an identifying section 21 uses the 
personal behavior table to create a "specific person table 24", which it then uses 
to detect if a person included in the specific person table 24 is present among 
persons in the field of view. (See Abstract of the Disclosure; as well as 
paragraphs [0002], [0012], and [0013].) Thus, as noted in paragraph [0013], 
Watanabe provides "an identifying section for searching for a specific person on 
the basis of the record items recorded in the personal behavior table" and "a 
detecting section for detecting a person for whom information is recorded in the 
specific person table, from a surveillance image". 

As is apparent from the foregoing brief description, both the overall 
purpose and the structural or methodological details of the Watanabe patent 
differ from the present invention, which is unrelated to identifying the presence 
or absence of specific individuals, and focuses on identifying certain behavior 
patterns which can be characterized as "abnormal" by tracking the movement of 
point features within the image, as described previously. 
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Watanabe is thus an example of the kinds of systems described at page 2, 

lines 8-13 of the present application. It utilizes complex algorithms for object 

recognition, tracking and alarm decision rules. It recognizes specific people by 

their image in the video surveillance data, and monitors their activity. Thus, as 

illustrated, for example, in Figure 5, the Watanabe system monitors complete 

images of humans and tracks and records the actions of each human in 

respective records. (See, for example, paragraphs [0068] -[0069].) 

For this purpose, Watanabe analyzes both the "attitude" and "behavior" of 
individuals depicted in the image. The term "attitude" refers to whether the 
subject is standing, sitting, bending, etc. (paragraph [0076]), and is determined 
based on a vertical/horizontal ratio or on a bounding rectangle. (See paragraphs 
[0076] -[0082].) The "behavior of a person, on the other hand, is determined as 
"stationary", "moving in a certain direction" and possibly including an indication 
of speed of motion. The attitude recognition function acts to identify an attitude 
(e.g., standing, bending, etc.) by the aspect ratio given by relative X and Y 
dimensions of the human image, and stored histograms representing those 
positions. By comparing human region images to the stored aspect ratios and 
histograms, the attitude of an image person may be deduced. The behavior 
recognition function identifies start and end positions of a human region, and 
speed of motion. The speed may be categorized, for example, into "stationary", 
"walk", "run" and the like, as discussed at paragraphs [0072] - [0075]. See also 
paragraphs [0084] - [0086]. 
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Watanabe thus operates only to observe the movement and positions of 

humans in an image region, and stores the observed attitudes, behavior and 

identifying features of each human in a table specific to that human, for the 

purpose of subsequently identifying the individuals who are present in an image, 

and distinguishing between those individuals and individuals who have not 

previously appeared in the image. That is, the Watanabe system simply 

identifies known individuals who appear in an image, based on certain 

previously observed behavior and attitude patterns. 

Accordingly, there is no suggestion in Watanabe of implementing a 
learning function, in which "normal" behavior of track point features at any 
position of the image is determined, as disclosed and claimed in the present 
application. That is, according to the present invention as defined in Claim 1, 
when a point feature passes through a position behaving differently from the 
learned normal behavior, than that behavior can be determined as "abnormal" 

"'V 

and an alarm or other action may be activated as appropriate, as recited, for 
example in Claim 1. 

The Office Action indicates that Watanabe discloses a "classification 
device receiving both the signal representing the normal range of behavior of the 
tracks and the data representing the tracks, being adapted to compare the signal 
and the data to issue a normal/abnormal signal in accordance with the outcome 
of such comparison" referring to reference numeral 81 in Figure 15. In this 
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regard, however, Applicants note that reference numeral 81 designates a 

"detecting section", which does not function to classify observed behavior as 

"normal" or "abnormal". Rather, it functions to detect a specific person identified 

to it by an identifying section 21. The detecting section 81 "detects the specific 

person recorded in the specific person table 82, by means of the specific person 

detecting section 32". (See paragraph [0156].) As noted in paragraph [0111], the 

purpose of this action is to determine whether a person contained within an 

image matches the characteristics of a person in the specific person table 24. 

The result of such a determination is simply to identify a particular individual 

depicted in the image. It is unrelated to whether the behavior of that individual 

is "normal" or "abnormal", or to generate an alarm signal to warn of abnormal 

behavior. 

The Office Action also indicates that Watanabe includes a learning device 
which receives data representing "the tracks" and produces "a signal 
representing a range of behavior considered normal by the learning device" in 
response to the operation of a learning process, referring in particular to 
reference numeral 64 in Figure 16. Applicants note, however, that the reference 
numeral 64 in Figure 16 designates a "behavior record creating section" which 
does not amount to a learning device. Rather, it merely stores information 
describing observed actions of particular humans, as indicated, for example, at 
paragraph [0152]. No learning is involved. 
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Finally, Applicants respectfully submit that paragraph [0149], which is 

considered as describing "an alarm generation device", does not in fact refer to 

alarms generated in response to the indication of a track which is judged to be 

"abnormal". Indeed, no "abnormal" tracks are defined in Watanabe. Rather, 

paragraph [0149] refers to sending search criteria from a client section 60 to a 

central server section 90, so that the latter may search its person table to 

identify one or more specific persons who match the criteria sent to it by client 

section 60. Accordingly, Applicants respectfully submit that independent Claims 

1 and 15, and all claims depending therefrom distinguish over Watanabe and are 

allowable. 

In addition, to the foregoing, Applicants respectfully submits that the 
following features of the invention as recited in the claims noted below are also 
missing in Watanabe, and in the other cited references as well: 

• The learning process accumulates data representing behavior of the 
tracks over a period of time in a four-dimensional histogram, with 
the four dimensions representing respectively, x-position, x-velocity, 
y -position, and y- velocity within the video image. (Claim 3) 

• Data concerning mobile tracks are stored in a four-dimensional 
histogram, while data concerning stationary tracks are stored in a 
two-dimensional histogram. (Claim 4) 
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A cell size of the four-dimensional histogram varies in accordance 

with measured speed of an image of each respective track. (Claim 

5) 

The histogram is periodically "de-weighted" in order to bias the 
result of the learning process towards more recent events. 

The comparison process classifies a track according to "a 
comparison of the frequency of occupation of a corresponding 
histogram cell with an occupancy threshold." (Claim 7) 

The comparison process acts to classify as normal behavior a track 
adjacent or near a cell which is above the occupancy threshold, 
despite the track's appearing in a cell which is below the occupancy 
threshold. (Claim 8) 

The abnormal tracks are filtered based on various criteria. (Claims 
9-12) 

Subsequent active alarm signals are inhibited for a predetermined 
time interval after a first active alarm signal. (Claim 13); and 

Subsequent active alarm signals are inhibited if caused by an 
abnormal track within a predetermined distance of another track 
which has previously generated an alarm. (Claim 14) 
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In regard to the above distinctions, the Schwerdt et al article describes 
equipment for identifying and tracking the position of faces within a visual 
image, and identifying facial expressions on the identified faces. As mentioned 
at the foot of page 42, a two-dimensional histogram is used, with luminance- 
chrominance dimensions, plotting "learnt" skin tones, to obtain a probability of 
whether a certain pixel represents skin (top of page 43). Moreover, as noted in 
the second full paragraph of page 43, and illustrated in Fig. 2, an identified 
region of skin tone is analysed to give it a point position and a spatial extent 
representing the size of the skin region. The location may be weighted with a 
Gaussian function to raise the importance of likely skin tone regions near a point 
at which a face is to be expected. 

In section 3.2, on pages 46-47, four- and five-dimensional histograms are 
discussed. The histograms have dimensions corresponding to eigenvectors 
derived from received image data, and the eigenvectors are those found most 
useful in distinguishing facial expressions. In defining the histograms, a number 
of known images, training samples, are used to derive the eigenvectors to use 
and to define the expression detected by a certain combination of eigenvectors. 
Thus, any received image or video data can be analysed by colour to detect a face. 
The face may then be reduced to a point position and an area, for the purpose of 
tracking its position. The image of the face region (fig. 2) may be analysed to 
derive eigenvectors, which may be compared to a 4D histogram to determine 
what expression the imaged face is wearing. 
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Thus, in Schwerdt et al, as in the present invention, a point feature is 

tracked. However, in schwerdt et al the point is tracked to enable the system to 

apply the expression recognition methods to the face region only. Unlike the 

present invention, there is no concept of learning normal behavior for the tracked 

point object. 

Unlike Schwerdt et al (and Watanabe) the present invention provides that 
a number of tracked point objects will be used to build up a histogram storing 
normal behavior: x-velocity and y-velocity for a track passing through any image 
pixel: x-position, y-position. This observed behavior is summarized in a 4D 
histogram. When a later tracked point object shows a certain x-velocity and y- 
velocity in a certain x-position and y-position, this will be compared to the 
histogram cell corresponding to those four dimensions. If the histogram cell has 
a population lower than a certain value, then the behavior of that track will be 
deemed abnormal. 

There is no suggestion of such abnormality detection in Schwerdt et al, 
which discloses the use of training images only to populate a histogram in order 
to determine the expression on an imaged face. 

CLAIMS 3 and 4 

Applicants acknowledge that, as indicated in the Office Action, Schwerdt 
et al shows a 4D histogram. Such 4D histograms are not new. The histogram in 
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Schwerdt et al, however, segregates image data in order to determine the facial 

expression worn by an imaged face. Watanabe attempts to identify an individual 

and store data representing that person's actions in a location specific to that 

person. When the same person returns to an imaged area, Watanabe seeks to 

recognize that person, either automatically or by user action, and to perform 

some action appropriate to the person appearing in the image, based on 

accumulated knowledge of their previous behavior. Watanabe does not seek to 

classify tracked paths as "normal" or "abnormal" in any part of a received image; 

rather it builds up a store of past behavior of a number of individuals, associates 

the behavior with the specific person, and uses this store of previous behavior to 

determine whether that individual is treated in a particular way the next time 

they are detected in the images region. Watanabe does not store data 

representing "normal" movement in order to identify "abnormal" movement 

when it occurs, so there can be no suggestion of storing such data for Watanabe 

in the 4D histogram of Schwerdt et al. 

Claim 7 

The Office Action indicates that Schwerdt et al shows a method of 
classifying a track according to the frequency of occupation of the corresponding 
histogram cell with a threshold. Schwerdt et al does not, however, disclose the 
use of an occupancy threshold in conjunction with the 4D histogram. In Table 2 
and the surrounding description, it is clear that a 4D histogram is formed, based 
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on the four eigenvestors which best serve to distinguish facial expressions. The 

similarity of an input image with earlier training images, summarised in the 

histogram, provides an estimate of the expression viewed. No occupancy 

thresholds are disclosed or suggested. Mention is made of empty histogram cells 

reducing a reliability of detection, but not of deciding whether observed behavior 

is normal or abnormal, as recited in the present invention. 

Claim 13 

The Office Action refers to Kamin as disclosing inhibition of active alarm 
signals for a predetermined time period after a first active alarm signal has been 
produced. Applicants respectfully submit, however, that Kamin rather describes 
blocking of spurious alarm indications caused by a change in overall brightness. 
The blocking attempts to block all alarm indications - not just ones following a 
first active alarm. The blocking function in Kamin operates for as long as there 
is a general brightness variation in the observed image, not for a predetermined 
time. 

Finally, the Wyschogrod et al patent has been cited in respect of Claim 14, 
as disclosing a method of video surveillance in which subsequent active alarm 
signals are inhibited if caused by an abnormal track within a predetermined 
distance of another track which has previously generated an alarm. The cited 
passage appears to refer to a feature which can disable the generation of 
multiple alarms if an arriving aircraft ("target") is ambiguously projected onto 
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more than one runway. That is, a single incoming aircraft may cause one of two 

or more incursions: arrival on one of two or more projected runways. The 

passage referred to in the Office Action (Column 106, line 23) allows a user to set 

the alarm system to generate a single alarm in such circumstances, rather than 

generating separate alarms related to each of the runways which may receive 

the incoming aircraft. Accordingly, insofar as Applicants have been able to 

determine, this portion of the disclosure in Wyschogrod et al does not teach or 

suggest inhibiting an alarm signal if "it is caused by an abnormal track within a 

predetermined distance of another track which has previou sly generated an 

alarm ". Notably, the alarm referred to in the cited passage of Wyschogrod et al 

relates to future events, while the alarm of Claim 14 of the present application 

relates to detective tracks (that is, past events). 

In light of the foregoing remarks, this application should be in 
consideration for allowance, and early passage of this case to issue is respectfully 
requested. If there are any questions regarding this amendment or the 
application in general, a telephone call to the undersigned would be appreciated 
since this should expedite the prosecution of the application for all concerned. 

If necessary to effect a timely response, this paper should be considered as 
a petition for an Extension of Time sufficient to effect a timely response, and 
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please charge any deficiency in fees or credit any overpayments to Deposit 

Account No. 05-1323 (Docket #038819.55861US). 



Respectfully submitted, 




Gary R. Edwards 
Registration No. 31,824 
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